{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import SVG\n",
    "from sklearn.datasets import load_digits\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dataset\n",
    "- digits dataset in scikit-learn\n",
    "- url: http://scikit-learn.org/stable/auto_examples/datasets/plot_digits_last_image.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_data = data.images\n",
    "y_data = data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# flatten X_data\n",
    "X_data = X_data.reshape(X_data.shape[0], X_data.shape[1]*X_data.shape[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1797, 64)\n",
      "(1797,)\n"
     ]
    }
   ],
   "source": [
    "# shape of data\n",
    "print(X_data.shape)\n",
    "print(y_data.shape)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Functional API\n",
    "- Creating models by Sequential API is easy and simple, but it is impossible to create complicated model structures\n",
    "- For instance, inception or residual net structure is impossible to implement using Sequential API since they require operations such as layer merging and multiple outputs\n",
    "- In this case, one could take advantage of Functional API\n",
    "    - Create model by defining inputs and outputs\n",
    "\n",
    "<br>\n",
    "<img src=\"http://www.deeplearningmodel.net/img/googlenet/googlenet_block.png\" style=\"width: 300px\"/>\n",
    "<center> **Inception Module in GoogleNet** </center>\n",
    "\n",
    "<img src=\"http://cv-tricks.com/wp-content/uploads/2017/03/600x299xResNet.png.pagespeed.ic.M1J-VkbWPB.png\" style=\"width: 300px\"/>\n",
    "<center> **Resudiual Structure in ResNet** </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single input & output\n",
    "- Model with only single input & output\n",
    "- Such structure is able to create using Sequential API as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# creating layers\n",
    "input_layer = Input(shape = X_data.shape[1:])\n",
    "activation_1 = Activation('relu')(input_layer)\n",
    "hidden_layer = Dense(50)(activation_1)\n",
    "activation_2 = Activation('relu')(hidden_layer)\n",
    "output_layer = Dense(10, activation = 'softmax')(activation_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# creating model\n",
    "model = Model(inputs = input_layer, outputs = output_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                3250      \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 3,760\n",
      "Trainable params: 3,760\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merging layers\n",
    "- Sometimes, it is necessary to merge layers (e.g., GoogleNet or ResNet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. concatenate\n",
    "- concatenate() simply merges results of two or more layers\n",
    "- For instance, assume there are two layers to be concatenated, whose results are\n",
    "**[x1, x2, ..., xn]** and **[y1, y2, ..., yn]**. Then, concatenated layer would be **[x1, ..., xn, ..., y1, ..., yn]**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(?, 50)\n",
      "(?, 50)\n",
      "(?, 100)\n"
     ]
    }
   ],
   "source": [
    "# creating layers\n",
    "input_layer = Input(shape = X_data.shape[1:])\n",
    "activation_1 = Activation('relu')(input_layer)\n",
    "hidden_layer_1 = Dense(50, activation = 'relu')(activation_1)\n",
    "hidden_layer_2 = Dense(50, activation = 'relu')(activation_1)\n",
    "concat_layer = concatenate([hidden_layer_1, hidden_layer_2])\n",
    "print(hidden_layer_1.shape)\n",
    "print(hidden_layer_2.shape)\n",
    "print(concat_layer.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. add, subtract, multiply, average, maximum\n",
    "- Such layers perform element-wise operations over all corresponding elements of two or more layers\n",
    "- Hence, dimensionality of the input layers are preserved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(?, 50)\n",
      "(?, 50)\n",
      "(?, 1)\n"
     ]
    }
   ],
   "source": [
    "# creating layers\n",
    "input_layer = Input(shape = X_data.shape[1:])\n",
    "activation_1 = Activation('relu')(input_layer)\n",
    "hidden_layer_1 = Dense(50, activation = 'relu')(activation_1)\n",
    "hidden_layer_2 = Dense(50, activation = 'relu')(activation_1)\n",
    "add_layer = add([hidden_layer_1, hidden_layer_2])\n",
    "print(hidden_layer_1.shape)\n",
    "print(hidden_layer_2.shape)\n",
    "print(add_layer.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. dot\n",
    "- dot() performs inner product operation between two layer results\n",
    "- 'axes' should be defined to perform the operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(?, 50)\n",
      "(?, 50)\n",
      "(?, 1)\n"
     ]
    }
   ],
   "source": [
    "# creating layers\n",
    "input_layer = Input(shape = X_data.shape[1:])\n",
    "activation_1 = Activation('relu')(input_layer)\n",
    "hidden_layer_1 = Dense(50, activation = 'relu')(activation_1)\n",
    "hidden_layer_2 = Dense(50, activation = 'relu')(activation_1)\n",
    "dot_layer = dot([hidden_layer_1, hidden_layer_2], axes = -1)\n",
    "print(hidden_layer_1.shape)\n",
    "print(hidden_layer_2.shape)\n",
    "print(dot_layer.shape)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
